{
 "cells": [
      {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/gimseng/99-ML-Learning-Projects/blob/master/003/solution/digit_recog_nn.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST HANDWRITTEN DIGIT RECOGNITION\n",
    "\n",
    "- The dataset was constructed from a number of scanned document dataset available from the National Institute of Standards and Technology (NIST). This is where the name for the dataset comes from, as the Modified NIST or MNIST dataset.\n",
    "\n",
    "- Images of digits were taken from a variety of scanned documents, normalized in size and centered. This makes it an excellent dataset for evaluating models, allowing the developer to focus on the machine learning with very little data cleaning or preparation required.\n",
    "\n",
    "- Each image is a 28 by 28 pixel square (784 pixels total). A standard split of the dataset is used to evaluate and compare models, where 60,000 images are used to train a model and a separate set of 10,000 images are used to test it.\n",
    "\n",
    "- It is a digit recognition task. As such there are 10 digits (0 to 9) or 10 classes to predict. Results are reported using prediction error, which is nothing more than the inverted classification accuracy.\n",
    "\n",
    "#### In this notebook you will learn the MNIST handwritten digit recognition problem and deep learning models developed in Python using the Keras library that are capable of achieving excellent results.\n",
    "\n",
    "#### Working through this tutorial you will learn:\n",
    "\n",
    "- How to load the MNIST dataset in Keras and generate plots of the dataset.\n",
    "- How to reshape the MNIST dataset and develop a simple but well performing multi-layer perceptron model on the problem.\n",
    "- How to use Keras to create convolutional neural network models for MNIST.\n",
    "- How to develop and evaluate larger CNN models for MNIST capable of near world class results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Artificial Neural Network (ANN)\n",
    "\n",
    "An artificial neural network is a mathematical model that converts a set of inputs to a set of outputs through a number of hidden layers. An ANN works with hidden layers, each of which is a transient form associated with a probability. In a typical neural network, each node of a layer takes all nodes of the previous layer as input. A model may have one or more hidden layers.\n",
    "\n",
    "![NN structure](images/ANN.jpg)\n",
    "\n",
    "ANNs receive an input layer to transform it through hidden layers. An ANN is initialized by assigning random weights and biases to each node of the hidden layers. As the training data is fed into the model, it modifies these weights and biases using the errors generated at each step. Hence, our model “learns” the pattern when going through the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "warnings.simplefilter('ignore')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.layers import Dropout\n",
    "from keras.layers import Flatten\n",
    "from keras.layers import BatchNormalization\n",
    "from keras.layers.convolutional import Conv2D\n",
    "from keras.layers.convolutional import MaxPooling2D\n",
    "from keras.utils import np_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We'll use the keras library for training the model in this tutorial. Keras is a high-level library in Python that is a wrapper over TensorFlow, CNTK and Theano. By default, Keras uses a TensorFlow backend by default, and we’ll use the same to train our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import mnist\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAATsAAAD7CAYAAAAVQzPHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAXUUlEQVR4nO3de2wV1fYH8O8SxRcRKGqtgIBJQfEXFEVEL1EUMFzUgOKLgEAk1gQwaNCAXjQaX/hMfICCyEsIeBNEUEOE1AIxYgP4uBeopUgCFhsQFUFRucD6/dFxM3vsaU/PmTMz5+zvJ2nO2rN7zqxr113MzJmHqCqIiArdCXEnQEQUBTY7InICmx0ROYHNjoicwGZHRE5gsyMiJ2TV7ERkkIhUi8h2EZkSVlJEcWNtFx7J9Dw7EWkBYBuAgQBqAWwAMFxVt4aXHlH0WNuF6cQs3tsbwHZV3QEAIrIEwBAAKQtCRHgGc3LsU9Wz4k4ioZpV26zrRElZ19nsxrYH8J1vXOsto/ywM+4EEoy1nb9S1nU2W3bSwLK//QsnImUAyrJYD1HUmqxt1nX+yabZ1QLo6Bt3APB98JdUdRaAWQA39ylvNFnbrOv8k81u7AYApSLSRURaArgTwIpw0iKKFWu7AGW8ZaeqR0RkAoCPAbQAMEdVt4SWGVFMWNuFKeNTTzJaGTf3k2STqvaKO4lCwLpOlJR1zSsoiMgJbHZE5AQ2OyJyApsdETmBzY6InMBmR0ROYLMjIidkc7kYERWoyy67zBpPmDDBxKNGjbLmFixYYOLXXnvNmvviiy9ykF1muGVHRE5gsyMiJ7DZEZETeG1sA1q0aGGNW7dunfZ7/cc2TjvtNGuuW7duJh4/frw19+KLL5p4+PDh1twff/xh4mnTpllzTzzxRNq5BfDa2JDkS1035pJLLrHGn3zyiTU+44wz0vqcX375xRq3a9cuu8Saj9fGEpHb2OyIyAkFferJeeedZ41btmxp4quuusqa69u3r4nbtGljzQ0bNiyUfGpra0386quvWnM333yziQ8ePGjNff311yZeu3ZtKLkQ9e7d28RLly615oKHbvyHu4L1efjwYRMHd1v79Olj4uBpKP73RYFbdkTkBDY7InICmx0ROaHgTj3xf4Ue/Pq8OaeQhOHYsWPW+O677zbxr7/+mvJ9dXV11vjnn382cXV1dUjZ8dSTsCT51BP/6U+XXnqpNbdw4UITd+jQwZoTsZ8m6e8TwWNvzz//vImXLFmS8nOmTp1qzT377LON5p4hnnpCRG5jsyMiJxTcqSe7du0y8Y8//mjNhbEbW1lZaY33799vja+99loTB79af+edd7JeP1FzzJw508TBK3MyFdwdbtWqlYmDp0b169fPxD169Ahl/Znilh0ROYHNjoicwGZHRE4ouGN2P/30k4kfeugha+7GG2808ZdffmnNBS/f8vvqq69MPHDgQGvut99+s8YXXXSRiSdOnJhGxkThCd5h+IYbbjBx8HQSv+Cxtg8++MAa++/K8/3331tz/v8v+U+TAoDrrrsurfVHocktOxGZIyJ7RWSzb1mRiKwWkRrvtW1u0yQKH2vbLensxs4DMCiwbAqAclUtBVDujYnyzTywtp2R1hUUItIZwIeq+n/euBpAP1WtE5ESAGtUtVsjH/HX58R6prn/BoTBOzf4v6IfO3asNTdy5EgTL168OEfZRY5XUCCc2o67rhu7aqixm26uXLnSxMHTUq655hpr7D9tZPbs2dbcDz/8kHIdR48eNfGhQ4dSriPEB/OEfgVFsarWAYD3enammRElDGu7QOX8CwoRKQNQluv1EEWJdZ1/Mt2y2+Nt4sN73ZvqF1V1lqr24i4T5Ym0apt1nX8y3bJbAWA0gGne6/LQMsqhAwcOpJwLPijE75577jHxu+++a80F72xCeS/xtd21a1dr7D/FKnhJ5L59+0wcvJvO/PnzTRy8C89HH33U6DgTp556qjWeNGmSiUeMGJH15zclnVNPFgNYD6CbiNSKyFjUF8JAEakBMNAbE+UV1rZbmtyyU9VUVw/3DzkXokixtt1ScFdQZOrxxx83cfAsdP9X5AMGDLDmVq1aldO8iADg5JNPNrH/agYAGDx4sImDp1SNGjXKxBs3brTmgruVUQs+ECvXeG0sETmBzY6InMBmR0RO4DE7j//uJf5TTQD7Upa33nrLmquoqLDG/uMi06dPt+aifLgRFZaePXua2H+MLmjIkCHWmA9VP45bdkTkBDY7InICd2Mb8O2331rjMWPGmHju3LnW3F133ZVyfPrpp1tzCxYsMHHwbHaixrz88ssmDt4E07+rmrTd1hNOOL49FffVRtyyIyInsNkRkRPY7IjICTxml4Zly5aZuKamxprzH0sBgP79j19W+cwzz1hznTp1MvHTTz9tze3evTvrPKlw+B8OBdh3Iw6ewrRixYpIcsqE/zhdMG//g6yiwC07InICmx0ROYHNjoicwGN2zbR582ZrfPvtt1vjm266ycTBc/LuvfdeE5eWllpzwYdvk9uCt19q2bKliffute8UH7x7dtT8t5/y3yotKPjks4cffjhXKTWIW3ZE5AQ2OyJyAndjs7R//35r/M4775g4+DDhE088/p/76quvtub69etn4jVr1oSXIBWcP//80xpHfemhf7cVAKZOnWpi/8N/AKC2ttbEL730kjUXfMhPrnHLjoicwGZHRE5gsyMiJ/CYXTP16NHDGt96663W+PLLLzex/xhd0NatW63xunXrQsiOXBDH5WH+y9WCx+XuuOMOEy9fbj9TfNiwYblNrBm4ZUdETmCzIyIncDe2Ad26dbPGEyZMMPEtt9xizZ1zzjlpf+7Ro0dNHDxdIO67uFKyBO9G7B8PHTrUmps4cWLo63/ggQes8aOPPmri1q1bW3OLFi0ysf+h3EnDLTsickKTzU5EOopIhYhUicgWEZnoLS8SkdUiUuO9ts19ukThYW27JZ0tuyMAJqnqhQD6ABgvIt0BTAFQrqqlAMq9MVE+YW07pMljdqpaB6DOiw+KSBWA9gCGAOjn/dp8AGsATM5JljkQPNY2fPhwE/uP0QFA586dM1qH/4HZgH134iTfXdYVSa7t4F19/eNg7b766qsmnjNnjjX3448/mrhPnz7WnP9JeBdffLE116FDB2u8a9cuE3/88cfW3IwZM/7+PyCBmnXMTkQ6A+gJoBJAsVcsfxXN2WEnRxQV1nbhS/vbWBFpBWApgPtV9UDw26JG3lcGoCyz9IhyL5PaZl3nn7SanYichPpiWKSq73mL94hIiarWiUgJgL0NvVdVZwGY5X2ONvQ7uVJcXGyNu3fvbuLXX3/dmrvgggsyWkdlZaU1fuGFF0wcPJucp5ckT6a1HWddt2jRwhqPGzfOxMErFg4cOGDi4A1jG/PZZ59Z44qKChM/9thjaX9OkqTzbawAeBtAlar6H6W1AsBoLx4NYHnwvURJxtp2Szpbdv8AcBeA/4rIX88+ewTANAD/FpGxAHYBuC03KRLlDGvbIel8G/spgFQHMfqnWE6UeKxtt+T95WJFRUXWeObMmSb236kBAM4///yM1uE/fhG822rwa/jff/89o3UQ+a1fv94ab9iwwcT+O+sEBU9LCR639vOflrJkyRJrLheXoMWNl4sRkRPY7IjICRI8UzunK8vwK/orrrjCGvtvHti7d29rrn379pmsAocOHTKx/4x0AHjmmWdM/Ntvv2X0+Qm0SVV7xZ1EIYji1JOSkhIT+58/DNgPvAmeI+j///crr7xizb3xxhsm3r59eyh5JkDKuuaWHRE5gc2OiJzAZkdETsiLY3bTpk2zxsEHfqQSfKjNhx9+aOIjR45Yc/5TSoIPvi5QPGYXkqgvF6NG8ZgdEbmNzY6InJAXu7GUE9yNDQnrOlG4G0tEbmOzIyInsNkRkRPY7IjICWx2ROQENjsicgKbHRE5gc2OiJzAZkdETmCzIyInRP3AnX0AdgI404uTwNVcOkW0Hhcksa6BZOUTVS4p6zrSa2PNSkU2JuW6TOZCYUna3y9J+SQhF+7GEpET2OyIyAlxNbtZMa23IcyFwpK0v1+S8ok9l1iO2RERRY27sUTkhEibnYgMEpFqEdkuIlOiXLe3/jkisldENvuWFYnIahGp8V7bRpRLRxGpEJEqEdkiIhPjzIeyE2dts67TE1mzE5EWAKYD+CeA7gCGi0j3qNbvmQdgUGDZFADlqloKoNwbR+EIgEmqeiGAPgDGe/894sqHMpSA2p4H1nWTotyy6w1gu6ruUNXDAJYAGBLh+qGq6wD8FFg8BMB8L54PYGhEudSp6hdefBBAFYD2ceVDWYm1tlnX6Ymy2bUH8J1vXOsti1uxqtYB9X8oAGdHnYCIdAbQE0BlEvKhZktibcdeR0mr6yibnTSwzPmvgkWkFYClAO5X1QNx50MZYW0HJLGuo2x2tQA6+sYdAHwf4fpT2SMiJQDgve6NasUichLqC2KRqr4Xdz6UsSTWNus6IMpmtwFAqYh0EZGWAO4EsCLC9aeyAsBoLx4NYHkUKxURAfA2gCpVfTnufCgrSaxt1nWQqkb2A2AwgG0AvgXwryjX7a1/MYA6AP9D/b/GYwG0Q/23QzXea1FEufRF/a7OfwB85f0Mjisf/mT994yttlnX6f3wCgoicgKvoCAiJ7DZEZETsmp2cV/+RZQrrO3Ck/ExO+8SmW0ABqL+oOgGAMNVdWt46RFFj7VdmLJ5BoW5RAYAROSvS2RSFoSI8NuQ5NinqmfFnURCNau2WdeJkrKus9mNTeIlMpS+nXEnkGCs7fyVsq6z2bJL6xIZESkDUJbFeoii1mRts67zTzbNLq1LZFR1FrxbMnNzn/JEk7XNus4/2ezGJvESGaIwsLYLUMZbdqp6REQmAPgYQAsAc1R1S2iZEcWEtV2YIr1cjJv7ibJJE/IA5XzHuk6UlHXNKyiIyAlsdkTkBDY7InICmx0ROYHNjoicwGZHRE5gsyMiJ7DZEZET2OyIyAlsdkTkBDY7InJCNrd4ohD179/fxIsWLbLmrrnmGhNXV1dHlhNROqZOnWriJ554wpo74YTj21P9+vWz5tauXZvTvIK4ZUdETmCzIyIn5MVu7NVXX22N27VrZ+Jly5ZFnU5OXH755SbesGFDjJkQNW7MmDHWePLkySY+duxYyvdFeTu5hnDLjoicwGZHRE5gsyMiJ+TFMbvgV9alpaUmztdjdv6v5AGgS5cuJu7UqZM1J9LQk/2I4hGsz1NOOSWmTJqHW3ZE5AQ2OyJyQl7sxo4aNcoar1+/PqZMwlNSUmKN77nnHhMvXLjQmvvmm28iyYkolQEDBpj4vvvuS/l7wVq98cYbTbxnz57wE2sGbtkRkRPY7IjICWx2ROSEvDhmFzxNoxDMnj075VxNTU2EmRD9Xd++fa3x3LlzTdy6deuU73vhhRes8c6dO8NNLAtNdhERmSMie0Vks29ZkYisFpEa77VtbtMkCh9r2y3pbDLNAzAosGwKgHJVLQVQ7o2J8s08sLad0eRurKquE5HOgcVDAPTz4vkA1gCYjBD16NHDxMXFxWF+dCI0tiuwevXqCDNxV1y1nQ9Gjx5tjc8999yUv7tmzRoTL1iwIFcpZS3Tg2HFqloHAN7r2eGlRBQr1naByvkXFCJSBqAs1+shihLrOv9kumW3R0RKAMB73ZvqF1V1lqr2UtVeGa6LKEpp1TbrOv9kumW3AsBoANO81+WhZeQZPHiwiU899dSwPz4W/mOP/rucBO3evTuKdKhhOa/tJDrzzDOt8d13322N/Xcg3r9/vzX31FNP5S6xEKVz6sliAOsBdBORWhEZi/pCGCgiNQAGemOivMLadks638YOTzHVP8VyorzA2nZLYq+g6NatW8q5LVu2RJhJeF588UUTB0+n2bZtm4kPHjwYWU7krs6dO5t46dKlab/vtddes8YVFRVhpZRThXcdFhFRA9jsiMgJbHZE5ITEHrNrTJIeIn3GGWdY40GDjl9qOXLkSGvu+uuvT/k5Tz75pImDX+0T5YK/Vv2XZzakvLzcxK+88krOcsolbtkRkRPY7IjICXm5G1tUVJTR+y6++GITB5/F6n+gSIcOHay5li1bmnjEiBHWXPDGor///ruJKysrrbk///zTxCeeaP+n37RpU6O5E2Vr6NCh1njatNTnS3/66afW2H8XlF9++SXcxCLCLTsicgKbHRE5gc2OiJyQ2GN2/mNfqmrNvfnmmyZ+5JFH0v5M/9frwWN2R44cMfGhQ4esua1bt5p4zpw51tzGjRut8dq1a00cfChwbW2tiYN3cuGDsCkXMr0kbMeOHdY47gdch4FbdkTkBDY7InICmx0ROSGxx+zGjRtn4uCDdq+66qqMPnPXrl0mfv/99625qqoqE3/++ecZfX5QWZn9iIKzzjrLxMFjIkS5MHny8Qej+e823JTGzsHLV9yyIyInsNkRkRMSuxvr99xzz8WdQkb69099d+/mnAZAlK5LLrnEGjd2px2/5cvt5wpVV1eHllNScMuOiJzAZkdETmCzIyIn5MUxu0K0bNmyuFOgArRq1Spr3LZt25S/6z/FasyYMblKKTG4ZUdETmCzIyIncDeWqIC0a9fOGjd21cSMGTNM/Ouvv+Ysp6RocstORDqKSIWIVInIFhGZ6C0vEpHVIlLjvaY+OECUQKxtt6SzG3sEwCRVvRBAHwDjRaQ7gCkAylW1FEC5NybKJ6xthzTZ7FS1TlW/8OKDAKoAtAcwBMB879fmAxja8CcQJRNr2y3NOmYnIp0B9ARQCaBYVeuA+qIRkbNDz67A+O+O3LVrV2surDutUGbyubbnzp1r4uDT7hrz2Wef5SKdxEq72YlIKwBLAdyvqgeCtzVv5H1lAMqa/EWimGRS26zr/JPWPwMichLqi2GRqr7nLd4jIiXefAmAvQ29V1VnqWovVe0VRsJEYcq0tlnX+afJLTup/2fubQBVqvqyb2oFgNEApnmvyxt4O/n4HxzUnN0Nyo18re3gnU38D3gPnmpy+PBhE0+fPt2aK4SH6DRHOrux/wBwF4D/ishX3rJHUF8I/xaRsQB2AbgtNykS5Qxr2yFNNjtV/RRAqoMYqW/YRpRwrG23cF+KiJzAy8VicuWVV1rjefPmxZMI5Z02bdpY43POOSfl7+7evdvEDz74YM5yygfcsiMiJ7DZEZETuBsboXRPxCai8HHLjoicwGZHRE5gsyMiJ/CYXQ6tXLnSGt92G0/Ep+x988031th/95K+fftGnU7e4JYdETmBzY6InCD+O3HkfGUi0a2MmrKJtycKB+s6UVLWNbfsiMgJbHZE5AQ2OyJyApsdETmBzY6InMBmR0ROYLMjIiew2RGRE9jsiMgJbHZE5ISo73qyD8BOAGd6cRK4mkuniNbjgiTWNZCsfKLKJWVdR3ptrFmpyMakXJfJXCgsSfv7JSmfJOTC3VgicgKbHRE5Ia5mNyum9TaEuVBYkvb3S1I+secSyzE7IqKocTeWiJwQabMTkUEiUi0i20VkSpTr9tY/R0T2ishm37IiEVktIjXea9uIcukoIhUiUiUiW0RkYpz5UHbirG3WdXoia3Yi0gLAdAD/BNAdwHAR6R7V+j3zAAwKLJsCoFxVSwGUe+MoHAEwSVUvBNAHwHjvv0dc+VCGElDb88C6blKUW3a9AWxX1R2qehjAEgBDIlw/VHUdgJ8Ci4cAmO/F8wEMjSiXOlX9wosPAqgC0D6ufCgrsdY26zo9UTa79gC+841rvWVxK1bVOqD+DwXg7KgTEJHOAHoCqExCPtRsSazt2OsoaXUdZbOTBpY5/1WwiLQCsBTA/ap6IO58KCOs7YAk1nWUza4WQEffuAOA7yNcfyp7RKQEALzXvVGtWEROQn1BLFLV9+LOhzKWxNpmXQdE2ew2ACgVkS4i0hLAnQBWRLj+VFYAGO3FowEsj2KlIiIA3gZQpaovx50PZSWJtc26DlLVyH4ADAawDcC3AP4V5bq99S8GUAfgf6j/13gsgHao/3aoxnstiiiXvqjf1fkPgK+8n8Fx5cOfrP+esdU26zq9H15BQURO4BUUROQENjsicgKbHRE5gc2OiJzAZkdETmCzIyInsNkRkRPY7IjICf8PU6S97J2P81QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot 4 images as gray scale\n",
    "plt.subplot(221)\n",
    "plt.imshow(x_train[0], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(222)\n",
    "plt.imshow(x_train[1], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(223)\n",
    "plt.imshow(x_train[2], cmap=plt.get_cmap('gray'))\n",
    "plt.subplot(224)\n",
    "plt.imshow(x_train[3], cmap=plt.get_cmap('gray'))\n",
    "# show the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 28, 28)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning Data\n",
    "\n",
    "Now that we have seen the structure of the data, we are ready to create the model.\n",
    "To work with the Keras API, we need to reshape each image to the format of (M x N x 1). We’ll use the .reshape() method to perform this action. Finally, normalize the image data by dividing each pixel value by 255 (since RGB value can range from 0 to 255):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# input color dimentions\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "num_pixels = img_rows * img_cols\n",
    "\n",
    "x_train = x_train.reshape((x_train.shape[0], num_pixels)).astype(float)\n",
    "x_test = x_test.reshape((x_test.shape[0], num_pixels)).astype(float)\n",
    "\n",
    "# normalize inputs from 0-255 to 0-1\n",
    "x_train /= 255\n",
    "x_test /= 255\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let us now convert the label integers into binary class using ``` to_categorical() ``` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from keras.utils import to_categorical\n",
    "\n",
    "num_classes = 10\n",
    "\n",
    "y_train = np_utils.to_categorical(y_train)\n",
    "y_test = np_utils.to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Designing the model\n",
    "\n",
    "This is the most important step in this notebook as it the accuracy is determined by how well we have defined the model.\n",
    "For this tutorial, we have used [Keras Documentation design](https://keras.io/examples/vision/mnist_convnet/) \n",
    "\n",
    "Keras provides us with two different types of model, first the Sequential model (which we'll be using in this tutorial) and the other being Functional Model.\n",
    "The sequential API allows you to create models layer-by-layer for most problems. Alternatively, the functional API allows you to create models that have a lot more flexibility as you can easily define models where layers connect to more than just the previous and next layers.\n",
    "\n",
    "For further learning purpose : Read [this](https://medium.com/@hanify/sequential-api-vs-functional-api-model-in-keras-266823d7cd5e) & [this](https://www.deeplearningdemystified.com/article/pdl-1) (I personally find this one really interesting and useful)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# baseline model : Neural Network\n",
    "def nn_model():\n",
    "    \n",
    "    # create model\n",
    "    nn = Sequential()\n",
    "    nn.add(Dense(num_pixels, activation = 'relu', kernel_initializer = 'normal', input_dim = num_pixels))\n",
    "    nn.add(Dense(128, activation = 'relu', kernel_initializer = 'normal'))\n",
    "    nn.add(Dense(num_classes, activation = 'softmax', kernel_initializer = 'normal'))\n",
    "    \n",
    "    # compile model\n",
    "    nn.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "    \n",
    "    return nn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense (Dense)                (None, 784)               615440    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 128)               100480    \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 717,210\n",
      "Trainable params: 717,210\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/10\n",
      "469/469 - 4s - loss: 0.2516 - accuracy: 0.9272 - val_loss: 0.1076 - val_accuracy: 0.9696\n",
      "Epoch 2/10\n",
      "469/469 - 5s - loss: 0.0879 - accuracy: 0.9732 - val_loss: 0.0874 - val_accuracy: 0.9714\n",
      "Epoch 3/10\n",
      "469/469 - 5s - loss: 0.0529 - accuracy: 0.9836 - val_loss: 0.0694 - val_accuracy: 0.9763\n",
      "Epoch 4/10\n",
      "469/469 - 5s - loss: 0.0363 - accuracy: 0.9893 - val_loss: 0.0728 - val_accuracy: 0.9770\n",
      "Epoch 5/10\n",
      "469/469 - 5s - loss: 0.0253 - accuracy: 0.9921 - val_loss: 0.0632 - val_accuracy: 0.9788\n",
      "Epoch 6/10\n",
      "469/469 - 5s - loss: 0.0216 - accuracy: 0.9930 - val_loss: 0.0644 - val_accuracy: 0.9806\n",
      "Epoch 7/10\n",
      "469/469 - 5s - loss: 0.0161 - accuracy: 0.9947 - val_loss: 0.0740 - val_accuracy: 0.9810\n",
      "Epoch 8/10\n",
      "469/469 - 5s - loss: 0.0147 - accuracy: 0.9949 - val_loss: 0.0719 - val_accuracy: 0.9816\n",
      "Epoch 9/10\n",
      "469/469 - 5s - loss: 0.0146 - accuracy: 0.9955 - val_loss: 0.0582 - val_accuracy: 0.9839\n",
      "Epoch 10/10\n",
      "469/469 - 5s - loss: 0.0089 - accuracy: 0.9970 - val_loss: 0.0845 - val_accuracy: 0.9797\n",
      "Test loss: 8.45\n",
      "Test accuracy: 97.97\n"
     ]
    }
   ],
   "source": [
    "batch_size = 128\n",
    "epochs = 10\n",
    "\n",
    "# build the model\n",
    "model = nn_model()\n",
    "\n",
    "print(model.summary())\n",
    "\n",
    "# Fit the model\n",
    "model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=epochs, batch_size=batch_size, verbose=2)\n",
    "\n",
    "# Final evaluation of the model\n",
    "scores = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', round(scores[0]*100, 2))\n",
    "print('Test accuracy:', round(scores[1]*100, 2))\n",
    "model.save(\"test_nn.h5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convoluted Neural Networks (CNN)\n",
    "\n",
    "In a CNN, the images are represented as 3-D array with the 3rd dimension containing the color(RGB). Consequently, a node of the hidden layer would only be connected to a small region in the vicinity of the corresponding input layer, making the process far more efficient than a traditional neural network. CNNs, therefore, are popular when it comes to working with images and videos.\n",
    "\n",
    "![Basic CNN MNIST Model](images/cnn-procedure.png)\n",
    "\n",
    "The various types of layers in a CNN are as follows:\n",
    "- **convolutional layers** : these run input through certain filters, which identify features in the image\n",
    "- **pooling layers**: these combine convolutional features, helping in feature reduction\n",
    "- **flatten layers**: these convert an N-dimentional layer to a 1D layer\n",
    "- **classification layer**: the final layer, which tells us the final result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data\n",
    "(X_train, Y_train), (X_test, Y_test) = mnist.load_data()\n",
    "\n",
    "# reshape to be [samples][width][height][channels]\n",
    "X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1).astype('float32')\n",
    "X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1).astype('float32')\n",
    "\n",
    "\n",
    "# normalize inputs from 0-255 to 0-1\n",
    "X_train = X_train / 255\n",
    "X_test = X_test / 255\n",
    "\n",
    "# one hot encode outputs\n",
    "Y_train = np_utils.to_categorical(Y_train)\n",
    "Y_test = np_utils.to_categorical(Y_test)\n",
    "\n",
    "num_classes = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define CNN model : Neural network with CNN\n",
    "\n",
    "def cnn_model():\n",
    "    \n",
    "    # create model\n",
    "    cnn = Sequential()\n",
    "    cnn.add(Conv2D(30, (5, 5), input_shape=(img_rows, img_cols, 1), activation='relu'))\n",
    "    cnn.add(MaxPooling2D())\n",
    "    cnn.add(Conv2D(15, (3, 3), activation='relu'))\n",
    "    cnn.add(MaxPooling2D())\n",
    "    cnn.add(Dropout(0.2))\n",
    "    cnn.add(BatchNormalization())\n",
    "    cnn.add(Flatten())\n",
    "    \n",
    "    cnn.add(Dense(128, activation = 'relu'))\n",
    "    cnn.add(Dense(64, activation = 'relu'))\n",
    "    cnn.add(Dense(num_classes, activation='softmax'))\n",
    "    \n",
    "    # compile model\n",
    "    cnn.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])\n",
    "    \n",
    "    return cnn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 24, 24, 30)        780       \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 12, 12, 30)        0         \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 10, 10, 15)        4065      \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 5, 5, 15)          0         \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 5, 5, 15)          0         \n",
      "_________________________________________________________________\n",
      "batch_normalization (BatchNo (None, 5, 5, 15)          60        \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 375)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 128)               48128     \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 64)                8256      \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 10)                650       \n",
      "=================================================================\n",
      "Total params: 61,939\n",
      "Trainable params: 61,909\n",
      "Non-trainable params: 30\n",
      "_________________________________________________________________\n",
      "None\n",
      "Epoch 1/10\n",
      "469/469 - 29s - loss: 0.1859 - accuracy: 0.9430 - val_loss: 0.0740 - val_accuracy: 0.9790\n",
      "Epoch 2/10\n",
      "469/469 - 26s - loss: 0.0620 - accuracy: 0.9805 - val_loss: 0.0336 - val_accuracy: 0.9888\n",
      "Epoch 3/10\n",
      "469/469 - 30s - loss: 0.0467 - accuracy: 0.9857 - val_loss: 0.0357 - val_accuracy: 0.9883\n",
      "Epoch 4/10\n",
      "469/469 - 26s - loss: 0.0393 - accuracy: 0.9878 - val_loss: 0.0407 - val_accuracy: 0.9866\n",
      "Epoch 5/10\n",
      "469/469 - 29s - loss: 0.0336 - accuracy: 0.9892 - val_loss: 0.0297 - val_accuracy: 0.9897\n",
      "Epoch 6/10\n",
      "469/469 - 31s - loss: 0.0288 - accuracy: 0.9907 - val_loss: 0.0293 - val_accuracy: 0.9908\n",
      "Epoch 7/10\n",
      "469/469 - 28s - loss: 0.0261 - accuracy: 0.9915 - val_loss: 0.0287 - val_accuracy: 0.9908\n",
      "Epoch 8/10\n",
      "469/469 - 28s - loss: 0.0239 - accuracy: 0.9923 - val_loss: 0.0285 - val_accuracy: 0.9914\n",
      "Epoch 9/10\n",
      "469/469 - 27s - loss: 0.0228 - accuracy: 0.9925 - val_loss: 0.0263 - val_accuracy: 0.9922\n",
      "Epoch 10/10\n",
      "469/469 - 26s - loss: 0.0202 - accuracy: 0.9935 - val_loss: 0.0334 - val_accuracy: 0.9910\n",
      "Test loss: 3.34\n",
      "Test accuracy: 99.1\n"
     ]
    }
   ],
   "source": [
    "batch_size = 128\n",
    "epochs = 10\n",
    "\n",
    "# build the model\n",
    "model = cnn_model()\n",
    "\n",
    "print(model.summary())\n",
    "\n",
    "# Fit the model\n",
    "model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=epochs, batch_size=batch_size, verbose=2)\n",
    "\n",
    "# Final evaluation of the model\n",
    "scores = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test loss:', round(scores[0]*100, 2))\n",
    "print('Test accuracy:', round(scores[1]*100, 2))\n",
    "model.save(\"test_cnn.h5\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python3.6Test",
   "language": "python",
   "name": "python3.6test"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
